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Abstract: This paper proposes a model for implementing a college-wide initiative designed 
to promote student competence with essential learning outcomes. This mixed-methods 
study combined descriptive and qualitative approaches to explore experiences of students, 
faculty, and professional as they engaged in a project that focused on teaching and 
learning with essential learning outcomes (ELOs). The researchers relied on the use of 
descriptive statistics and interpretive phenomenological analysis to capture faculty and 
professional staff participants ’ engagement in a collaborative study group designed to 
support pedagogy integrating ELOs into courses, assignments, and co-curricular activities 
and to gauge students ’ experienced competence. Six themes emerged from the focus on 
ELO integration, including connection, awareness, utility, reflection, intention and 
facilitation. Additionally, using a pre and post student survey of self-perceived ELO 
competence, researchers found a difference in student ratings of 0.25, (p < 0.05), 
suggesting an increase in students ’perception of their ELO competence over the course of 
the semester. 
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Introduction 

Many college and universities have responded to accrediting organizations’ requirements 
for student outcomes assessment by establishing institutional goals. This requirement, coupled 
with employers’ demands for colleges and universities to prepare graduates with practical skills, 
professional knowledge, and intellectual abilities, has created challenges for higher education in 
the United States. To assist liberal arts institutions establish assessment practices and collection of 
evidence for direct assessment of student learning that satisfy both accreditors and employers, the 
American Association of Colleges and Universities (AAC&U) launched its Liberal Education and 
America’ s Promise (LEAP) initiative in 2005 (https://www.aacu.org/leap) . Since then, over 100 
liberal arts colleges and universities have adopted LEAP essential learning outcomes 
(https://www.aacu.org/leap/essential-leaming-outcomes) and its associated Valid Assessment of 
Learning in Undergraduate Education (VALUE) rubrics (https://www.aacu.org/value) . Although 
informed by AAC&U’s leadership in outcomes assessment, the public liberal arts college in this 
study identified ten essential learning outcomes (ELOs) specific to curricular and co-curricular 
learning experiences offered on its campus. Some of the resulting ELOs overlap with LEAP’S, 
however, not all. As well, this college embraced institution-wide participation in the design and 
implementation of its ELO initiative rather than creating an administratively driven compliance 
model to address accreditors’ requirements and employers’ demands. This approach has led to 
gradual, but increasing, campus-wide commitment to the ELO initiative; however, it has also 
necessitated ongoing professional development. The study described in this article presents the 
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findings from one professional development opportunity and offers a model for learner-centered 
implementation of ELOs in courses, assignments, and co-curricular activities. 


Background 


Essential Learning Outcomes 

Shifting the current model of education to one that includes a learner-centered, 
competency-based approach to education is an important priority for refonn in the 21 st century. 
(Altbach, Reisberg, & Rumbley, 2010; Barber, Donnelly, & Rizvi, 2013). While there continues 
to be a growing emphasis on quality pedagogy, colleges and universities must also create 
cultures that value student learning outcomes in their efforts to ensure institutional effectiveness 
(Kuh, Jankowski, Ikenberry, & Kinzie, 2014). LEAP’S essential learning outcomes (Figure 1) 
represent one national project supporting this model of outcomes-focused and competence-based 
student learning. 

The LEAP National Leadership Council (2007) refers to a "Framework for Excellence," 
which encompasses the process of integrating ELOs across disciplines through an intentional focus 
on the part of teachers and students. As part of the LEAP initiative, AAC&U also promotes the 
importance of “High Impact Practices” to support the students’ acquisition of essential learning 
outcomes. The council emphasizes the importance of offering students opportunities to apply 
interdisciplinary knowledge and to acquire a perspective for learning that involves creating 
knowledge through real-world problem-solving activities. 



The Essential 



Knowledge of Human Cultures and the Physical and Natural World 
★ Through study in the sciences and mathematics, social sciences, humanities. 


histories, languages, and the arts 
FotustJ t'v engagement with big question!. b 


Intellectual and Practical Skills, including 


★ Inquiry and analysis 

★ Critical and creative thinking 


★ Quantitative literacy 

★ Information literacy 


* Written and oral communication ★ Teamwork and problem solving 

Praclitei extensively, across the turns ulum, in the context of progressively iti 


Personal and Social Responsibility, including 
Ik Civic knowledge and engagement—local and global 

★ Intercuttura! knowledge and competence 

★ Ethical reasoning and action 

Ik Foundations and skills for lifelong learning 
Anduntd through active involvement with diverse comma 



Integrative and Applied Learning, including 

it Synthesis and advanced accomplishment across general and specialized studies 
DemonstrMeJ through the application of knowledge, skills, nul responsibilities to 


Figure 1. LEAP (2005) Essential Learning Outcomes. 
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Conceptual Models of Instruction that Focus on ELOs and Supporting Student Competence 

Supporting student learning outcomes and the characteristics of instruction and assessment 
tasks that foster competency-based abilities in students requires careful consideration of the 
models of learning and assessment employed to promote that education. Former models of 
competency-based approaches to learning focused on delivery of instruction in and testing for 
discrete skills. Since the model proposed by Voorhees (2001), however, competency-based 
learning has been conceived as the interplay of student learning outcomes, objectives, skills and 
competencies. He proposed (Voorhees, 2001) the unambiguous identification of competencies and 
student learning outcomes and an emphasis on reflection as a way to integrate meaningful learning 
and foster students’ awareness about the criteria for evaluation. Competencies, according to 
Voorhees (2001) are acquired throughout students’ undergraduate education; as a result, they cut 
across disciplines, majors, and general education. 

Voorhees’ (2001) model includes an emphasis on linking and embedding competencies in 
courses across disciplines. This offers the opportunity for teachers to intentionally reinforce 
competencies in course syllabi and course assignments. Even so, students need the criteria used to 
evaluate their level of competency in course tasks. Intentional instruction helps students increase 
their knowledge of the skills and abilities they develop as well as their understanding of the 
“integrative learning experiences in which skills, abilities and knowledge interact to fonn 
learning,” (Voorhees, 2001, p.9). 

Voorhees (2001) argues that students leam best when they are aware of outcomes expected 
from their course work and that faculty members need to know what they are trying to achieve 
with each of the assignments they ask students to complete. Voorhees’ (2001) model creates the 
opportunity for teachers and students to collaborate on the identification of specific ELOs for 
courses and assignments. 

More recently, Shavelson (2007, p. 12) offers a conceptual framework for measuring 
student learning that integrates research on cognition and illustrates the need for students to 
transfer domain-specific knowledge to problem-solving contexts outside of that domain using 
broad abilities and critical thinking. Shavelson’s (2007) framework identifies abilities that closely 
parallel LEAP’S ELOs. In addition, Shavelson (2007) posits the universality of the abilities and the 
need for learning situated within a meaningful context for students, which he argues (Shavelson, 
2007), leads to active engagement, deliberate practice, and fonnative feedback, all of which 
contribute to students’ abilities to transfer the learning to new situations. 

MacLean and Scott (2011) propose a ten-step framework for fostering competencies. After 
a (1) needs analysis, they argue that (2) course aims and learning outcomes need to be clearly 
stated from the outset so that learners have an understanding of the teacher’s expectations. They 
assert that doing so not only organizes the course, but also facilitates the learning process and 
supports student understanding of their processes of change. With a (3) clear outline of the course 
structure and (4) course content, students’ interest and personal goals can be aligned with 
intellectual and professional capabilities. Learning outcomes must be (5) clearly indicated and 
learning experiences should be relevant, timely, interesting. The teacher should provide regular, 
fonnative feedback to support student self-assessment. Students need to be (6) adequately 
supported, through formative feedback, and (7) assessment strategies should be transparent with 
candid criteria for evaluation. Modules for learning need to be (8) explicitly planned and (9) 
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implemented effectively with an (10) emphasis on evaluation. This model, although developed for 
an online environment, is a meaningful framework for face-to-face classroom instruction as well. 

A competency-based educational model proposed by Kovacs, Hutchinson, Collins, and 
Linde (2013) focuses on outcomes and levels of competence developed through students engaging 
in and performing real-world tasks. The model includes a three-step process in which educational 
goals are stated as competencies students acquire through course learning and assessment tasks. 
Step two requires procedures for assessing achievement of competencies, and step three focuses on 
the learning experiences that build competencies in students. 

The Science of Teaching and Learning: A New Paradigm for Teaching and Learning 

In a survey conducted by the National Institute for Learning Outcomes Assessment, 
researchers found that assessment efforts centered on yielding meaningful measures of student 
learning and documenting student accomplishment are most valued by provosts, rather than those 
measures that interest the government or accreditors (Kuh, Jankowski, Ikenberry, & Kinzie, 2014). 
Walker (2008) eloquently described a phenomenon that requires careful consideration of the 
unintended, critical impact of a model for learning that has prevailed, possibly as a result of a 
bureaucratic structure in which the institution of education exists, 

“over the centuries, we have refined our definitions of learning to mean a certain kind of 
school learning, and educational systems have been geared to nourish a narrow range of 
human talent which, when defined as “the ability to manipulate the abstractions of academe 
(p.12), is estimated to be one-tenth of human ability,” (Cross, 1976, p.12; Taylor, 1968, as 
cited in Walker, 2008). 

However, efforts toward a competency-based approach were evidence of a change in the way we 
approach learning and assessment in the classroom, one that can be thought of as a “New 
Paradigm for Teaching and Learning” (Barr & Tagg, 1995). Barr and Tagg describe a model of 
education that is focused on the learner rather than the instructor. The model asserts to improve the 
quality of learning through powerful learning environments in which learners are encouraged to 
discover information through learning experiences designed to support student competencies. 
Learning is constructed by the student and is shaped by individuals’ experiences. The criteria for 
success integrate quantity as well as quality student learning outcomes. Learning structures are 
integrated and holistic and are centered on the student through active engagement with the content. 
Through this model, ultimately the degree equals demonstrated knowledge and skills. Using this 
model, students have the opportunity to document quality learning through the significant learning 
experiences (Fink, 2013) that demonstrate student competence with specified essential learning 
outcomes. Artifacts then serve as quality learning and evidence of that competence. A focus on the 
construction of significant learning experiences (Fink, 2013) offers a model for instructors to 
support student competence by designing instruction that resides within the learning paradigm. 

Significant Learning Experiences 

Fink (2013) identifies taxonomy for significant learning and a model for integrated course 
design as part of a learner-centered approach to instruction. The taxonomy includes six 
components upon which the design for instruction is based. Foundational knowledge is part of the 
content which students need to leam, an essential part of any course and important to the program 
outcomes students need with any degree. Application serves as a second component in which the 
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learner applies content through integration, which is identified as a third component of the 
taxonomy. This component offers students the opportunity to integrate learning through engaging 
in authentic meaningful tasks that are personally relevant to students as part of a focus on 
integration of knowledge and abilities for depth of understanding. Through that engagement in 
authentic meaningful tasks, students acquire a commitment to the learning, which Fink describes 
as caring, a fourth component of the taxonomy. Fink refers to the human dimension as a fifth 
component, and acknowledges the role that the learner’s perspective plays in tenns of the learning 
experience as well as the perspective of others. Lastly, learning how to learn describes a 
component of the taxonomy that focuses on the learner’s awareness of the skills and abilities they 
have acquired through the learning experience. This awareness on the part of students is an 
important consideration as it relates to supporting student competence with essential learning in 
higher education. 

The Role of the Student: Student Awareness and Perceptions of Learning 

Student self-awareness of learning has the potential to support student competence with 
essential learning outcomes. The learner’s self-awareness of their learning can be described as 
metacognition. Metacognition is defined by Flavell (1976) as, “one’s knowledge concerning one’s 
own cognitive processes or anything related to them.” Metacognitive awareness can become a 
powerful learning strategy for students and has the potential to further support life-long learning 
(Wehlburg, 2006), what one might also consider essential learning outcomes. With metacognitive 
awareness of ELO competence, students can potentially acquire a greater understanding of the 
skills they possess and engage in cognitive thought processes to apply those skills in the life and 
workplace experiences they encounter. 

Relying on student perception of metacognitive awareness can play an important role in 
supporting students with developing ELO competence. In a study of student perceptions of 
learning, Walker (2008) found that student perceptions of learning revealed learning well beyond 
traditional course content. Students identified areas of competence that parallel those identified by 
LEAP as essential learning outcomes. In this sense, student reflection demonstrates validity as a 
direct measure of student learning related to essential learning outcomes. Recognizing the 
important role of student perception of learning has the potential to enhance and support other 
direct measures of student learning. Other direct measures of learning, such as performance based 
tasks, offer the opportunity for students to reflect, analyze and interpret concepts acquired and 
facilitate awareness of essential learning outcomes that go beyond the content of the course 
(Walker). 

Spronken-Smith et al. (2012) conducted a study in which they surveyed students as to 
which course learning and assessment activities students perceived fostered learning on various 
levels and recommended more structured opportunities for reflection to support student learning. 
This serves as evidence of the value of teachers who intentionally focused on informing students 
of the range of course outcomes and supporting student awareness of their own learning to foster 
deeper learning processes. 

Encouraging students to reflect and develop self-awareness of the Essential Learning 
Outcomes targeted in their coursework has the potential to support competency and the readiness 
needed for success in the 21 st century and a global economy. Engaging students in the practice of 
attending to the learning acquisition and promoting opportunities to develop metacognition may 
further support students’ ELO awareness as they develop strategies for life-long learning. Given 
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the value of ELO competence and the role that student awareness of ELO competence has in 
preparing students for post-secondary arena, this study served as a valuable opportunity to assess 
students’ level of ELO competence acquired over the course of the semester while enrolled in a 
class in which the instructor focused on ELO integration. Furthermore, students’ awareness of 
ELO competence was assessed through a written statement in which students communicated 
perceived competence in the ELOs targeted for instruction. 

The Research Study 

Study Design 

The design of this study is largely descriptive and qualitative in nature and included the use 
of Interpretive Phenomenological Analysis (IPA), a process of analysis used to identify the 
phenomena that emerge, recognizing the role of the researcher’s interpretation of the data (Reiners, 
2012). Methodology surrounding IPA includes interpreting data and uncovering thematic aspects 
of the experiences investigated in the study. Researchers are not independent of the meanings 
extracted from the data, but instead are part of the phenomena. In this study, the researchers sought 
to explore the phenomena surrounding students’, faculty members’, and professional staff 
members’ experiences integrating ELOs into courses, assignments, and co-curricular activities. 
While faculty and professional staff conducted their courses and co-curricular activities, 
researchers sought to identify themes that emerged from their experiences. The descriptive portion 
of the study involved pre and post surveys of students’ perceived competence with the ten ELOs 
identified by the college ( http://www.stockton.edu/elos. Appendix A). 

Setting/Participants 

A public liberal arts college in a suburban area was the venue for data collection. The 
college enrolls approximately 8,570 and offers baccalaureate and graduate level programs in the 
liberal arts and professional studies. The majority of undergraduates are full-time, matriculated 
students. 99% are from the state where the college is located. 75% are Caucasian. The male to 
female ratio is 40% to 60%. 35% of the students receive Pell Grants, and 76% borrow at some 
point during their college education. The college has 304 full-time faculty members. Twice as 
many are associate as full or assistant professors, and two-thirds of the faculty members are 
tenured. The sampling method used to solicit participants was criterion-based, non-probability 
sampling. This type of sampling “requires that one establish the criteria, bases or standards 
necessary for units to be included in the investigation, one then finds a sample that matches these 
criteria” (Merriam, 1988, p. 48). After approval from the Institutional Review Board, the 
researchers sent out a call for faculty and professional staff participants through the College’s 
email system. The call described the study group project and encouraged interested faculty and 
professional staff members to write a letter of interest to participate in a study group for the 
purpose of learning to integrate ELOs into courses, assignments, and co-curricular activities. 
Nineteen faculty and professional staff members were selected. In addition to attending study 
group meetings, they volunteered to share student data with the study group facilitators (the 
researchers) and to attend a focus group at the end of the semester. The study group began during 
the mid-semester break in 2013 and extended through the spring semester 2014. 

During the mid-semester break, faculty and professional staff participants attended a four- 
hour workshop and two, shorter follow-up meetings. Participants read and discussed relevant 
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scholarship on essential learning outcomes, course and co-curricular design, and integrative 
learning. The goal of these meetings was to encourage participants to practice intentional revision 
of their pedagogical practices, to incorporate ELOs into their course and co-curricular goals, and to 
associate relevant ELOs with particular learning experiences. In addition, participants were asked 
to design one learning task intended to elicit one or more ELOs. The researchers collected data 
about the effectiveness of these meetings through participant interviews and focus groups and 
students’ reflections. 

Faculty and professional staff participants selected the courses and co-curricular 
experiences targeted for ELO integration. Learning experiences ranged from those aimed at first 
through fourth year courses and co-curricular activities, and general studies courses to courses in. 
business, social sciences, and nursing. Student participants included those enrolled in the courses 
taught by faculty participants or those who attended co-curricular activities facilitated by the 
professional staff participants. 

The researchers collected additional data of students’ perceived competence with ELOs. 
Students were given pre and post surveys for self-reporting. The researchers also collected 
qualitative data from students’ end-of-semester reflections. 

Data Collection Procedures 

Consent. Participants and students signed IRB-approved infonned consent forms, agreeing 
to take part in interviews and focus groups (participants) and to complete pre and post surveys 
(students). Five faculty participants in the ELO study group and their students consented. 
Professional staff participants in the ELO study group did not consent, primarily because the ELO 
study group occurred after their co-curricular activities had already taken place. As a result, they 
joined the ELO study group to gain skills for ELO implementation in future activities. In total, the 
researchers collected data from 111 students enrolled in seven courses. 

ELO Student Perception Survey. At the beginning of the semester, after explaining the 
study and obtaining written consent, researchers read aloud the “Administration Instructions - 
ELO Survey” (Appendix B) to students who chose to participate. Researchers distributed the 
Essential Learning Outcomes Survey (Appendix B) to students. The survey asked students to rate 
their perceived level of competence in each of Stockton’s ten ELOs using a 3-point Likert Scale. 
Levels of competence included (1) aware, (2) competent and (3) skilled. These data were collected 
as a pretest to determine students’ baseline perceived level of competency. As the semester 
progressed, students engaged in learning and assessment tasks designed by participants that were 
designed to elicit specific ELOs. 

At the end of the semester students completed the same survey. Post-test data were 
compared to pretest results and analyzed using the paired /-test to detennine changes over time and 
as a result of intentional ELO integration into courses and assignments. 

Student Written Statements. Researchers provided faculty participants with the writing 
prompt shown in Appendix C. In addition, students were prompted to reflect on the important ELO 
outcomes of the course and to respond to the following questions: 

“ What have you learned with regard to these learning outcomes? ” 

‘How is this learning important to your overall knowledge? ” 
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“How will it help you in the future? ” 

Faculty participants provided these questions to the students in their courses to capture 
written statements describing their perceived level of ELO competence at the end of the semester. 
A content analysis of this data involved coding and thematic analysis to identify themes relating to 
students’ metacognitive knowledge of their competence in ELOs. 

Faculty Inteiwiews. Researchers conducted individual interviews with faculty participants 
to investigate phenomena surrounding student awareness of ELO competence. Each faculty 
member was asked to consider the following question 

“How aware do you think students are about the ELOs they acquire from experiences both 
in and out of the classroom? ” 

These interviews were transcribed, and the researchers perfonned a content analysis of the 
interview data, which involved coding and thematic analysis to determine emergent themes. 

Data Analysis 

Thematic Analysis Procedure. Faculty interview data were transcribed and coded to 
identify emerging themes using the process of Interpretive Phenomenological Analysis. This 
process included a six-phase analysis of qualitative data as identified by Braun and Clarke (2006). 
During the first phase, researchers transcribing the data, and read through the data a number of 
times. In the second phase, researchers used HyperRESEARCH (2014) to generate initial themes. 
Then, they tentatively aligned passages from the interviews with relevant themes. In the third 
phase, researchers reviewed the alignments and made any necessary adjustments matching 
passages to themes. The researchers then reviewed the data to ensure consensus (fourth phase) and 
organized the themes into a meaningful visual representation (fifth phase). Finally, the researchers 
refined the specific components of each theme, clearly defined each theme, and identified the 
themes on the visual representation in such a way that the visual representation communicated the 
findings of the study group (sixth phase). 

Student Statement Content Analysis Procedure. The content analysis process used with the 
reflective statements written by students included, first, transcribing the reflective statements (first 
phase), and reading the statements a number of times (second phase). In the third phase, using N- 
Vivo (2008), researchers highlighted instances in which students identified learning related to any 
one of the ELOs. The researchers, then, reviewed the data to ensure consensus (fourth phase). 
Finally, compelling examples of each theme were extracted from the student data to support the 
findings of the study group (fifth phase). It was not necessary to construct a separate visual 
representation to communicate these findings, as these results did not require any modification of 
the previously constructed visual representation (sixth phase). 

Results 

ELO Student Perception Survey. Findings indicated that overall scores on the post-test 
were higher with a mean difference of 0.25 ip < .05). These results show increase in student 
perceived ELO competence over the course of the semester. The differences in the means for each 
ELO in each course showed increases ranging from 0.16 to 0.52 (Figure 2). Furthermore, the 
respective means of the ELOs targeted by faculty participants for integration into their respective 
courses showed an increase at a level of significance (Table 1). 
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STUDENT ELO SURVEY 
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Figure 2. Outcome of Student ELO Survey. 


Table 1 


Difference in Pre and Post Means for each ELO 



Cl 

CT 

PC 

CO 

IL 

TW 

GA 

AC 

QR 

ER 

Mean Diff 

0.49 

0.44 

0.3 

0.34 

0.3 

0.28 

0.52 

0.27 

0.16 

0.16 

t 

6.18 

7.09 

4.50 

4.10 

4.60 

3.68 

5.78 

3.80 

2.73 

2.19 

p 

0.00001 

0.00001 

1.70E-05 

7.80E-05 

1.20E-05 

0.000364 

0.00001 

0.000241 

0.007377 

0.0308 


Cl = Creativity and Innovation, CT = Critical Thinking, PC = Program Competence, CS = Communication Skills, IL= Information Literacy, 
TW = Teamwork, GA = Global Awareness, AC = Adaptation & Change, QR = Quantitative Reasoning, ER = Ethical Reasoning 


Themes Identified in Student Statement Data. The thematic analysis of student statements 
focused on references to ELOs as interpreted by researchers. Table 2 reflects the results which 
included between 10 and 36 references for each ELO targeted for instruction. 

Students cited many examples of growth in the areas of these ELOs. For example, students’ 
responses included references to the ability to think beyond the scope of the class as evidenced by 
the following statements. 

“I have learned how to think and create beyond the scope of the class to 
achieve an open-ended goal. ” 

“It gave some perspective to the theories and knowledge we have learned by 
applying it to real world scenarios. ” 
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Students also identified the need to integrate the use of critical thinking skills with other 
infonnation and settings as evidenced in the following examples. 

“I have learned not to take everything I read at face value, but rather to compare what I 
read to my own personal views and to consider why a particular culture feels a certain 
way about a topic. ” 


Table 2 

Frequency of References to ELO Competence in Student Statements 


ELO 

Increase in Mean 

Number of References 

Critical Thinking 

+.0.44 

36 

Teamwork 

+ 0.28 

28 

Global Awareness 

+ 0.52 

35 

Ethical Reasoning 

+ 0.16 

10 


“Learning these skills helps in any setting, academic or professional, and 
helped me to identify more than my own point of view and how to be able to 
adapt to my surroundings. ” 

The integrated role that teamwork plays in meeting with success in group settings is evidenced by 
the following statements. 

“I’ve learned how to work with a group and bounce ideas off of those around 
me to better my skills and what I’m working on. This is important because, in 
the field of Programming, people often work in groups to get a code done and 
it ’.s' crucial to make sure everyone is on the same page and has the same idea 
about things. If you can’t communicate clearly, you are not going to do a 
good job working with a group to get something done and doing something by 
yourself might be overwhelming. This helps me in the future because I will 
now be able to communicate myself with a group a little bit better and be able 
to get my ideas across better and have a better way of saying something is or 
isn 7 a good idea. ” 

Students also recognized the role of teamwork in leading and meeting challenges in the field as in 
the following examples. 

“...gaining knowledge on teamwork will make me a better team member 
and/or leader. ” 
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“ ....do my job independently and learn, evaluate, and react to new challenges 
in this field. ” 

Students described Global Awareness skills as those that include a value for understanding various 
perspectives and the ability to better understand and interact with people as a result of these new 
understandings: 

“This learning is important to my overall knowledge because every day, I am 
surrounded by people who are culturally different from me. Whether it is 
race, religion, or something else, I know that not everyone will have the same 
views and practices that I do. This creates a better understanding of the 
world around us, and the people that it consists of. It can be applied to many 
different situations, whether it is at school, at work, or just at the store. Being 
culturally competent can be beneficial throughout your life in many different 
aspects. ” 

“In an ever-growing culturally diverse society, it is important that I make 
myself aware of the nuances of cultural groups to eliminate any biases or 
prejudices that I may have toward them. ” 

As examples of the growth in the area of Ethical Reasoning, students connected this learning 
outcome to the pragmatic application of these skills as in the following examples. 

“Having these skills will give me the knowledge to find and use appropriate 
resources and the confidence to question practices not based on evidence. ” 

“It is important to express your feelings and view others to gain experience 
and knowledge not just your own interpretations. ’’ 

These examples of self-described ELO acquisition on the part of students serve as another measure 
of student learning and further evidence of increased achievement in ELO competency. 

Themes Identified in Faculty Data. As a result of the thematic analysis of the faculty 
interview data, six themes were identified from shared perspectives of faculty participants. The 
following are features of ELO integration that support student competence with ELOs: 

Facilitation. This theme includes a reference to faculty members’ comfort level or 
knowledge about ELO instruction and the role that teaching plays in supporting student 
competency with ELOs. This theme might also identify the need for faculty to guide or facilitate 
ELO acquisition in students through planned or incidental advising. 

Intention. Faculty participants identified a need or value for explicit emphasis of ELOs on 
the part of instructors to support student understanding. References to this theme include 
statements regarding repeated presentations of ELOs to students, revisiting the ELOs with students 
and/or providing continuity in discussing the ELOs with students. 

Connection. This theme includes instances in which students connect one of the ELO to 
either another ELO or possibly connect one ELO intellectually or experientially over time. For 
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example, appreciating ELO knowledge from the beginning of the semester to the end of the 
semester. It might also include students recognizing continuity or a recursiveness in one or more 
ELOs. Finally, this theme included statements referring to linking concepts in a manner that 
demonstrates one or more ELOs. These concepts might also include course content or concepts 
learned in pragmatic experiences. 

Utility. In regard to this theme, faculty referred to the intended use of the ELO, a value for 
the authentic purpose of the ELO in the future. It might also refer to preparation of and/or 
application of the ELO beyond the scope of the classroom. This theme included references to the 
ELOs and life-long learning. 

Reflection. According to Boud, Keogh, and Walker (1985, p. 19), “reflection in the context 
of learning is a generic tenn for those intellectual and affective activities in which individuals 
engage to explore their experiences in order to lead to new understandings and appreciations”. For 
this reason all references that coincide with this definition were identified within the theme of 
reflection. Additionally, references to this process of reflection and documentation of the behavior 
were also considered references to this theme. 

Awareness. This theme emerged from statements that illustrated students’ ability to identify 
one or more ELOs through a learning experience or a life experience, which heightened or 
contributed to recognition of ELO competence. This experience was interpreted as one that 
illuminated students’ abilities as it related to ELO competence. 

Themes Identified in Faculty Data. As a method for ensuring the validity of the themes that 
emerged from this data, the researchers asked faculty participants to review the themes and 
comment on the credibility of the researcher’s findings. Faculty participants were asked to review 
the six themes and the definitions identified by the researchers as outlined in the discussion above. 
As a result of this check of the validity of the data, faculty participants reported that the themes 
identified were either credible or very credible interpretations of the data. 

Findings 

This mixed methods research revealed that several components of ELO integration 
converge to support student competence. Overall, students showed significant growth in all ten 
ELOs. This finding was further confirmed by student qualitative statements to further probe the 
level competence as it related to the ELOs. The phenomena that emerged from the faculty 
interviews included six themes as instrumental to supporting student competence with ELOs. 

Faculty identified the importance of (1) facilitating ELO integration by developing a 
comfort level or knowledge base with ELO instruction and recognizing the role of faculty in 
guiding and facilitating learning through planned or incidental experiences to effectively support 
ELO acquisition in students. Faculty have the opportunity to facilitate this phenomenon through 
explicit emphasis and (2) intentionality of ELOs on the part of instructors through repeated 
presentations with these in mind to students, revisiting the ELOs with students, using ELO 
vocabulary and/or providing continuity in discussing the ELOs with students has the potential to 
support ELO acquisition in students and support metacognitive awareness.. This further supports 
the phenomenon of (3) connection as another theme that emerged from this study. As students 
engage in learning experiences to support ELOs, the findings suggest that student learning might 
benefit from a focus on making explicit connections among ELOs and opportunities to see 
relationships between them and the practical settings for which they are intended. 
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A valuable benefit of students’ acquisition of ELOs included a fourth theme that emerged 
in the findings which relates to an understanding of the (4) utility of the ELOs acquired. As 
students gain competence with any given ELO this has the potential to transcend learning to a level 
that includes a greater (5) awareness about the pragmatic abilities students are acquiring. While 
faculty recognized that students were not as cognizant about ELOs, awareness emerged as a 
component to ELO acquisition in which students acquire heightened recognition of ELO 
competence. Greater awareness of ELO abilities may contribute to what might be thought of as the 
ultimate level of ELO competence. The final theme of (6) reflection has the potential to serve 
students as they engage in the opportunity to intentionally attend to the learning acquired and 
articulate specific learning to support greater awareness. Reflection might occur in a variety of 
ways, but most importantly offers students the opportunity to document their ELO experiences and 
acquired levels of ELO competence. A final important finding identified by researchers as a result 
of the analysis of the data included a phenomenon related to the integrated nature of the themes 
that emerged. As the data was coded and individual themes emerged, researchers repeatedly 
recognized several different themes emerging from any one statement. Excerpts of faculty 
interview data are included in Appendix D. 


Discussion 

This study was designed to explore the phenomenon surrounding the integration of ELOs 
from the perspective of faculty and student participants at a small liberal arts college through an 
initiative to weave ELOs into the culture of the college. Researchers interviewed faculty 
participants and utilized methods that included interpretive phenomenological analysis (Reiners, 
2012) to identify themes that emerged from the experiences of faculty as the study progressed. 
ELO Study Group participants and co-facilitators discussed the research, theory and practice 
surrounding student learning outcomes, and co-facilitators shared examples of learning 
experiences designed to support student learning outcomes. Faculty participants engaged in small 
group discussion for the purpose of supporting pedagogical practice and ELO integration. 

As the findings suggest, six themes emerged from the content analysis of faculty 
participant data. These themes are reflected in Figure 3 and further defined in the discussion that 
follows. 

Each of the emerging themes identified in this study is further supported by concepts 
identified in the review of literature. Because the researchers in this study recognize, it is difficult 
to discuss any one theme without reference to one or more additional themes, we have chosen to 
highlight this by italicizing the identified theme each time it is addressed in the discussion that 
follows. 

The concept of connection focuses on students making associations among the ELOs and 
concepts over time, which may include prior to the course, through the duration of the course, and 
beyond the course. This aligns with Fink’s (2013) component of the Taxonomy of Significant 
Learning identified as engaging in authentic meaningful tasks that are personally relevant to 
students as part of a focus on integration of knowledge and abilities for depth of understanding. 
The concept of connection is also relative to the Learning Paradigm (Barr & Tagg, 1995) in which 
the learning structures are integrated, holistic, and are centered on the student through active 
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engagement with the content, shaped by students’ individual experiences. This concept of 
connection can also be illustrated in Shavelson’s (2007) model as the need for learning that is 
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Figure 3. Emerging themes of ELO Integration. 


highly situated within a meaningful context, learning that includes extensive engagement, 
deliberate practice and fonnative feedback to support students’ ability to transfer the learning to 
new situations. This concept is further supported by Voorhees’ model which emphasizes the 
importance of competencies being linked and embedded with courses across disciplines. This 
fonns the foundation for learning that students acquire upon which further learning is built. 
MacLean and Scott (2011) reference this concept in their model which emphasizes student 
understanding of the “processes of change” and parallel the theme of connection as students 
connect concepts related to competencies over time. 

The concept of awareness as an emerging theme illustrates the important role that 
metacognition has in ELO competence. Metacognitive awareness can become a powerful learning 
strategy for students. It has the potential to further support life-long learning (Wehlburg, 2006), 
and is therefore an important component of this model of ELO integration. Fink (2013) 
emphasizes the importance of learning how to leam. This is a good illustration of the role that 
metacognitive awareness plays in this model. Voorhees (2001) identifies the importance of making 
students aware and reflective about the criteria for evaluation to support competence in students. 
He asserts the importance of faculty unambiguously identifying student learning outcomes and 
articulating specific competences as faculty guide to facilitate ELO acquisition in students. 
Through planned or incidental advising, students have the opportunity to build metacognitive 
awareness and further develop ELO competence. MacLean and Scott (2011) also recognize the 
importance of facilitating the learning process through effective organization of the course that 
includes clearly stated learning outcomes, a concept also included as part of the model proposed by 
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Kovacs, Hutchingson, Collins, and Linde (2013) who identify specifying educational goals 
explicitly as part of their three step process to supporting competence. This also supports an 
intentional focus, another emerging theme in this study. Faculty harness an intentional stance as 
they engage in the explicit identification of learning outcomes and facilitate learning on the part of 
students. 

Through an intentional stance, faculty and students have the opportunity to further support 
learning through practices such as explicitly identifying in the syllabus, opportunities for 
supporting students’ competence with ELOs (Voorhees, 2001). MacLean and Scott (2011) also 
note the need for faculty to clearly state course aims and learning outcomes and provide formative 
feedback in an effort to support student learning. For example, as a student engages in a learning 
experience that may have been designed with the intention to support a particular ELO, the student 
might gain a greater awareness about the specific use of the ELO, or a greater awareness of the 
value for the authentic purpose of the ELO in the future, or purpose beyond the scope of the 
classroom. 

This has the potential to further support student understanding by supporting their 
awareness of the utility of a particular ELO. The theme of utility appears to parallel, what Fi nk 
(2013) identifies as the integration of the concepts among the course, other courses and students’ 
personal, social and/or work life. To support competence with an ELO, students engage in learning 
experience intended to have meaning surrounding the authentic context in which it is intended to 
be utilized (Voorhees, 2001). MacLean and Scott (2011) recognize the need for course content to 
be meaningful to the context, interest and personal goals of the learners and emphasized the need 
for learning experiences to be relevant. This concept of utility is further developed as students 
engage in and perform real-word tasks, an important part of a competency-based model for 
supporting student learning outcomes (Kovacs, Hutchinson, Collins, & Linde, 2013). Engaging 
students in meaningful, authentic, perfonnance-based tasks has shown to support student self- 
perceived competence (Cydis, 2014). 

As students engage in the significant learning experiences, facilitated by faculty and 
develop the awareness of the utility of any given ELO, students and faculty created opportunities 
for reflection to further support ELO competence. These opportunities for reflection can be 
intentional as well as unintentional. Students identified more effective learning experiences given 
more opportunities to intentionally engage in reflection (Spronken-Smith et ah, 2012). Through the 
emphasis and integration of meaningful learning and providing students with the criteria for 
evaluation, students have opportunities to build awareness and reflectiveness about their 
competence (Voorhees, 2001). The process of reflection offers students opportunities to rely on 
aptitude, natural endowments and abilities to facilitate the acquisition of student learning outcomes 
and effectively transfer the learning to new situations (Shavelson, 2007). This is also further 
supported by the human dimension in Fink’s (2013) taxonomy in which students have the 
opportunity to learn about oneself. As students engage in reflection they have the opportunity to 
leam more about the competence they have acquired. 

Conclusions 

The results of this study confirm important components of a model for ELO integration 
into courses, assignments, and co-curricular activities. While the components of this model for 
ELO integration are interconnected, each also has an independent role in teaching and learning 
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with ELOs. The components of the model that emerged through this study are: connection, 
awareness, utility, reflection, intention and facilitation. Students, faculty, and professional staff 
noted that learning increased when these components were present in course/co-curricular activity 
design and in assignment and activity descriptions. For instance, students enhance their familiarity 
with ELOs through exposure to the vocabulary and continued conversation about ELOs throughout 
the semester. However, students may not be cognizant about ELO integration as they engage in 
course work designed to support ELO competence unless their teacher intentionally discusses 
ELOs. Therefore, teachers can enhance student awareness by engaging in explicit conversations 
about ELOs, revisiting discussion about how learning experiences support ELO competence, and 
providing students with visuals that clearly articulate and describe ELO competence. It is 
important to note that a teacher’s intentional focus on ELOs does not by itself increase students’ 
ability to develop ELO competence. In the context of an intentional pedagogy, students must also 
engage in reflection and metacognitive awareness of ELO competence in order to most effectively 
develop ELOs. However, faculty and professional staff can engage in practices that are likely to 
lead to development of students’ ELOs such as explicit articulation of ELOs in the course/co- 
curricular activity syllabus, identification of relevant ELOs in learning tasks, and evaluation of the 
demonstration of ELO learning in assessments. Faculty and professional staff can also provide 
various opportunities for students’ reflection of ELO competence. Finally, students, faculty, and 
professional staff can work collaboratively to identify and illuminate examples of student ELO 
competence exhibited in samples of reflective writing, meaningful research and analysis, and 
projects drawn from real-world experiences. 

While each of these themes operate in a very interdependent manner, it is important to note 
that the relationship among them appears to have the potential to create a synergetic dynamic and 
ultimately support and facilitate the impact of the other components. As the findings suggest, it is 
difficult to separate emergent themes as phenomena that supports effective ELO integration and 
student development of ELOs. The researchers propose that there is an interdependent relationship 
among these concepts and the synergetic relationship among these features contribute to an 
effective model of ELO implementation in an educational setting such as the one described in this 
study. Furthermore, this might also include a recursive nature of ELO competence on the part of the 
students. 

This phenomena is consistent with Fink’s Taxonomy of Significant Learning (2013), in 
which the components of the taxonomy are interdependent and do not exist as a hierarchy, but 
rather as a relationship that is recursive in nature. Figure 3 represents the interdependent 
relationship among the phenomena that emerged in this study and were shown to contribute to 
ELO competence in students. Finally, the theme of reflection emerged and appears to integrate 
very meaningfully within this synergetic model for supporting ELO integration. 

Limitations 

Researcher participants in this study were solicited based on their interest in methods for 
effective integration of ELOs in courses, assignments, and co-curricular activities and the value 
they placed on the conceptual models outlined in the literature review. For this reason, researcher 
and participant biases may have played a role in the results of this study. In future studies, 
researchers could consider adopting control group procedures for ensuring the credibility of the 
themes that emerged participant data. Additionally, instructor bias should be considered in relation 
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to the effect it may have on the metacognitive awareness students develop. As instructors discuss 
the concept of ELO competence and the importance of promoting ELO competence it may have an 
effect on students’ perceived ELO competence and this factor should therefore be considered. 
Creating a control group might also help to mitigate any effect resulting from students’ desires to 
craft reflective statements, or respond to the ELO survey with a perception of increased ELO 
competence to please their teacher. 

The Hawthorne (Shuttleworth, 2009) effect may have played a role in the results of the 
post-test. At the end of the semester, when students completed the post-test, they were aware that 
their teacher was participating in a study group to investigate ELO integration into their course and 
assignments. Consequently, students’ knew that their teacher sought to improve their ELO 
competence. To avoid this effect, future studies that include a control group with students from 
courses not intentionally integrating ELOs into courses and assignments would offer researchers 
an opportunity to compare reflective data and to draw more confident conclusions about the 
effectiveness of the ELO integration model. 

Future Research 

In future research, the researchers intend to collect direct measures of student learning 
including artifacts demonstrating ELO competence. These artifacts could be analyzed to ascertain 
evidence of ELO competence and the extent to which student performance correlates with their 
self-perception of ELO competence. The researchers will need to create an assessment plan to 
evaluate the student artifacts. 

Further research is needed to learn the specifics of the intentional practices used by faculty 
to foster ELO competence in students; this aspect of participants’ classroom work was not 
investigated as thoroughly as necessary to fully explore the efficacy of the study group and the 
ELO integration model. In addition, the researchers need to develop an assessment plan to evaluate 
the model proposed in this study, including an analysis of the extent to which the themes are 
interdependent. The researchers also intend to conduct a content analysis of syllabi designed by 
participants for their communication about ELOs and teachers’ intentional ELO focus. 

Finally, the researchers aim to analyze the usefulness of future innovations to the 
construction of the model of ELO integration presented in this study and to the assessment of the 
model’s effectiveness as a means to increase students’ learning and ELO competence. The 
researchers might consider experimenting with the Degree Qualifications Profile (Lumina 
Foundation, 2013) as a way to connect outcomes students acquire at the course level to the 
outcomes identified on such a national initiative. The model presented here is flexible enough to 
integrate those innovations, especially those that align with the college’s ten ELOs and that 
promote the most successful student learning experiences. 
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